Skip to content

feat(agent): report hook liveness and aggregated hook errors - #220

Merged
geekgonecrazy merged 7 commits into
devfrom
feat/agent-hook-health
Oct 12, 2026
Merged

feat(agent): report hook liveness and aggregated hook errors#220
geekgonecrazy merged 7 commits into
devfrom
feat/agent-hook-health

Conversation

@graywolf336

@graywolf336 graywolf336 commented Sep 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Problem

Installed hooks can silently stop recording because their shell guard suppresses failures. A shared per-agent JSON health snapshot also loses updates under concurrent writers and lets one session's success hide another session's failure.

Benefits

  • Detect silent hook failures using observed dispatch outcomes.
  • Identify the failing session and parent instead of relying on one shared agent timestamp.
  • Retain concurrent hook observations without lost snapshot updates.
  • Diagnose parsing and transport failures, including during service outages.

Changes

  • Emit structured Rust tracing observations with event ID, timestamp, agent, verb, outcome, detail, session ID, and parent session ID.
  • Append immutable session operational events transactionally to the existing atomic.redb through the shared local/reactor service handlers. Retry deduplication preserves the original committed sequence; session-indexed reads are paginated.
  • Keep operational diagnostics independent of the running-turn provenance journal, so malformed input and post-completion failures remain recordable.
  • Replace hook-health.json writes with log-derived health. Status JSON exposes hook_health, hook_sessions, and hook_health_error; a successful child cannot clear a parent's failure.
  • Carry caller-resolved parent correlation explicitly across the typed RPC boundary. Missing session IDs remain unassigned diagnostics.
  • Preserve observations during service outages in immutable per-event spool files, replayed idempotently after recovery. Status can inspect pending diagnostics without misreporting silent agents from an incomplete log.
  • Continue reading and aggregating the bounded tail of legacy hook-errors.log, including multiline errors and normalized signatures.

Verification

  • Native parse/initialization failures, typed parsing/transport/recovery, and status JSON regressions pass.
  • Concurrent storage tests retain 640 events with duplicate retries and session-indexed pagination.
  • Overlapping local and real TCP gRPC dispatches retain all 24 child events; append retries and session-specific queries pass.
  • Independent CLI processes, concurrent tracing contexts, outage-spool replay, and parent/child failure isolation pass.
  • Workspace Clippy, formatting, documentation, and protobuf contract checks pass.
  • Workspace tests outside atomic-cli, CLI unit tests, and the other CLI integration suites pass locally. The existing provenance_rpc_sink fixture fails locally because it expects a sibling atomicd binary, while the current Reactor checkout builds atomic-reactor.
  • All eight CI checks passed for implementation commit 7dc2734, including Linux, macOS, and Windows tests. CI is running again for the documentation-only follow-up df7985e.

Documentation

Vault artifacts are excluded from this PR.

@graywolf336
feat(agent): report hook liveness and aggregated hook errors
0cf03c5 
`atomic agent status` answered "are the hooks installed?" and stopped.
It could not answer "are they still working?", and the difference is
the expensive one: the shell guard in atomic-agent::hooks ends in
`|| true`, so a hook that dies is indistinguishable from one that was
never installed, and a failing dispatch drops the turn silently.

In this repository, .atomic/hook-errors.log held 1784 lines over six
days and 21 sessions — 546 of them the same failure, "Provenance turn
N is not running" — with no command that would ever read it.

- Record every hook dispatch to .atomic/hook-health.json: per-agent,
  per-verb last_fired plus outcome. last_ok is preserved across a later
  failure, so "worked then broke" is distinguishable from "never
  worked". Written on both the dispatch and the parse-failure paths.
- Read a bounded tail of hook-errors.log and aggregate it by a
  signature that collapses session ids and turn numbers, turning 546
  near-identical lines into one actionable row. Continuation lines are
  folded into their entry: the plugin appends messages unescaped, and
  dropping the tail was losing the actual error.
- Surface both in `atomic agent status`, human and --json, plus a
  silent_agents list naming hooks that are installed but have never
  recorded. That list stays empty when no health file exists at all,
  so a freshly-upgraded repository is not accused.

The two signals are printed together because either alone misleads: a
clean error log proves nothing about whether the recorder is running,
and a live recorder proves nothing about what it failed to do earlier.
@graywolf336
fix(agent): make the hook-health record safe under concurrent writers
36dd365 
Several agents record into one repository at once, and the record is a
read-modify-write, so two things broke.

- The temp file had a fixed name. Concurrent writers shared it, so the
  loser's rename failed outright and the winner could publish bytes the
  other had interleaved into the same path. HookHealth::read treats an
  unparseable file as "no data", so either failure discarded the whole
  record rather than one entry. The temp name now carries pid and a
  counter.
- Nothing serialised the read-modify-write, so the last writer won and
  silently dropped what the others had just recorded. For a first-ever
  fire that leaves an agent missing from the record, which reads as
  "hooks installed but never recorded" — the false alarm this feature
  exists to raise. Guarded now with an fs2 flock on .atomic/, retried
  for up to 50ms and then proceeding unlocked, because a hook that hangs
  is worse than one that loses a timestamp. flock is released by the
  kernel on descriptor close, so a process dying mid-write cannot wedge
  the file.

Verified with 30 concurrent hook processes across 6 verbs: all 6
survive, no corruption, no stray temp files.
@graywolf336
Merge dev into feat/agent-hook-health
78e1ecb
@graywolf336
fix(agent): report current hook age across all dispatch paths
4f47135
@graywolf336

graywolf336 commented Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Updated against current dev and resolved the merge conflict.

  • Fixed liveness age to compare the last successful dispatch with the current time.
  • Moved the shared health reporter into atomic-agent so native hooks, typed agent receive events, and the local/gRPC service handler use the same health format.
  • Parsing, transport, validation, and initialization failures now remain visible, with prior successes preserved and recovery clearing the corresponding failure.
  • Added regression coverage that exercises the shared service both directly and over real gRPC.

Local validation passed: 22 health unit tests, 9 status tests, 3 integration tests, the new doctest, formatting, and cargo clippy --workspace -- -D warnings.

All eight CI checks now pass on 4f47135, including the full Linux, macOS, and Windows test suites: https://github.com/atomicdotdev/atomic/actions/runs/37949909148

@graywolf336 graywolf336 mentioned this pull request Oct 12, 2026
Merged
@geekgonecrazy
geekgonecrazy merged commit 73822ab into dev Oct 12, 2026
8 checks passed
@geekgonecrazy
geekgonecrazy deleted the feat/agent-hook-health branch October 12, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@geekgonecrazy geekgonecrazy Awaiting requested review from geekgonecrazy

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@graywolf336 @geekgonecrazy